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Problem, Purposes, and Objectives 

Before programs of vocational education can be improved and befor 
access to programs of high quality can be insured, adequate systems ai; 
techniques of program evaluation must be developed and put into opera- 
tion. Educators are recognizing that the role of evaluation is to 
design, collect, analyze, and report the data necessary for sound edu- 
cational decision-making but have not, for lack of adequate preparation 
taken the necessary steps to fully develop and operationalize effec- 
tive evaluation programs. It was with an awareness of the problems 
facing those who must evaluate and recognition of the vital contribu- 
tion that well-designed evaluations can make to improving vocational 
education programs that this institute was conducted. The institute 
was designed specifically to help equip persons in leadership positions 
with procedures and techniques needed for obtaining valid and reliable 
evaluative data. 

The institute was planned so as to focus on the following four 
major purposes or expected outcomes: (1) participants would learn 

additional knowledges and skills needed for improving vocational educa- 
tion evaluation, (2) participants would initiate the development of an 
evaluation plan having relevance for use within their agency or organi- 
zation, (3) task force groups would interact with the consultants, 
synthesize the papers presented and make recommendations regarding the 
most appropriate techniques of evaluation available, and (4) the 
institute staff would consolidate the various reports into a general 
guide on strategies and procedures of evaluation. 

To accomplish these outcomes, specific objectives were established 
as follows: 

1. To emphasize the contribution that well-designed evaluations 
can make to sound educational decision-making (program 
planning and program improvement). 

2. To identify program objectives specified in vocational 
education legislation and their relationship to evaluation. 

3. To recognize the basic requirements essential to program 
evaluation efforts at the local, state, and national level. 




1 



CL <D 



4. To review selected theoretical and operational approaches 

for evaluating vocational education programs at these levels. 

Procedures and Activities 



A program planning committee was established and used to help 
select the consultants and assist in finalizing the institute program. 
Preparation for the institute also included the purchase of selected 
references and the solicitation of many other references from various 
researchers and state departments of education. 

To accomplish the purposes and objectives established, a variety 
of activities were used to enrich the understandings and experiences 
of the participants during the one-week institute. Included were form- 
al lectures, informal talks, a D^mposiura, large and small group discus- 
sions, reaction and questioning panels, small task force assignments, 
and individual assignments. 

One-hundred qualified applicants and several alternates, repre- 
senting a wide variety of backgrounds and as many states as possible, 
were selected to attend. Ninety- two persons representing thirty-six 
states and having responsibilities for evaluating vocational education 
programs at either the local, area, state, or national level partici- 
pated in the institute. 

Conclusions and Recommendations 



Evaluation of the institute included assessment of participant 
satisfaction with the overall program, a summary of the plans of action 
developed by the participants, and a follow-up survey conducted eight 
months after the institute to assess outcomes in terms of participant 
activities since the institute. From an analysis of the data gathered 
on the various instruments, it was concluded that the institute was 
successful in accomplishing the goals which had been established. The 
combined efforts of the participants, consultants, and institute staff 
resulted in eight task group reports and a general guide for improving 
vocational education evaluation. 

The major recommenda ion made is that additional institutes with 
minor modifications be held throughout the country to meet the needs 
of others having important, responsibilities for program evaluation. 

Other recommendations included that: (1) consideration be given to 

holding separate conferences for beginners and those who have had 
considerable evaluation experience, (2) additional introductory mater- 
ials be sent to the participants prior to the institute, (3) less 
emphasis be placed on theoretical approaches and more on how to make 
practical application of appropriate evaluation techniques, (4) more 
tisue be devoted to the task force groups, and (S) the practice of 
having each participant develop a tentative plan of action be continued. 
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I. INTRODUCTION 



The Problem 



both the Vocational Education Act of 1963 and the Vocational Educa- 
tion Amendments of 1968 provide for, and in fact require, evaluation of 
vocational education programs. The Declaration of Purpose states in 
part that funds are authorized to . . . "improve existing programs of 
vocational education" . . . and that persons of all ages . . . "will 
have ready access to vocational training or retraining which is of high 
quality" .... 

Before existing programs can be "improved" and before access to 
programs of "high quality" can be insured, adequate systems and tech- 
niques of evaluation must be developed and put into operation. The 
use of quick and often highly subjective devices for appraising the 
quantity and quality of vocational programs will not suffice. Profes- 
sional educators are recognizing the importance and complexity of the 
evaluation process but have not yet taken the necessary steps to fully 
develop and operationalize any effective evaluation program. 

The infant status of evaluation may in part account for the many 
common shortcomings of past evaluative efforts. 

1. Most evaluative efforts have failed to provide valid and 
reliable information needed to support sound decision-making 
because of the following: 

a. Reports often contain only impressionistic information 

b. Many reports are almost devoid of hard data 

c. An over -re liance on anecdotal comments 

d. An over-reliance on subjective determinations 

2. Evaluations have focused almost entirely on the educational 
process - curricular organization, staff activities and 
qualifications, and physical facilities while ignoring 
program outcomes. Process evaluations by themselves cannot 
gauge program effectiveness. 

3. Evaluations have too often been an after thought, partial and 
sporadic rather than planned, thorough and continuous. 

Unfortunately, the status of evaluation in both vocational and 
general education today can be summarized as follows: 

1. There is a lack of adequate theory pertaining to the nature 
of evaluations which are needed to effectively accommodate 
educational programs. 

2. There is a lack of knowledge about decision-making processes 
and information requirements. 

3. There is a lack of appropriate evaluation instruments and 
procedures for gathering data. 

4. There is a lack of mechanisms for organizing, processing, 
and reporting evaluative information. 

5. There is a critical shortage of trained evaluators. 
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At was an awareness of the problems facing those who must evaluate 
and recognition of the important contribution that well-designed evalua- 
tions can make to improving vocational education programs that supported 
the need for conducting this institute. 

Purposes of the Instit i te 

This institute was designed specifically to help equipe persons in 
leadership positions with knowledges i*nd skills for improving vocational 
education evaluations. It was designed to provide insights into a rela- 
tively unexplored area, and to provide trainees with the techniques and 
procedures needed for obtaining valid and reliable evaluative data. 

The institute also focused on three other major purposes or 
expected outcomes. First, the efforts of the participants and consul- 
tants were directed toward the preparation of eight task group reports. 
Each group reviewed and synthesized the papers presented and other re- 
ferences available so as to reach a consensus on the strategies and 
techniques of program evaluation most appropriate for their area of con- 
cern. Second, these reports and other references were used by the in- 
stitute staff to prepare a general guide on strategies and procedures 
of evaluation that can be used by others responsible for evaluating pro- 
grams. Third, the participants developed tentative "plans of action" 
which were relevant for their own particular area and level of respon- 
sibi lity . 

Objectives 



The objectives stated in the original proposal were delimited and 
refined by the program planning committee and institute staff. These 
objectives were used as general guides for selecting the topics to be 
presented and discussed at the institute. The specific objectives estab- 
lished were as follows: 

1. To emphasize the contribution that well-designed evaluations 
can make to sound educational decision-making (program plan- 
ning and program improvement) . 

2. To identify program objectives specified in vocational 
education legislation and their relationship to evaluation. 

3. To recognize the basic requirements essential to program 
evaluation efforts at the local, state, and national level. 

4. To review selected theoretical and operational approaches for 
evaluating vocational education programs at these levels 
including : 

a. Determining data requirements 

b. Reviewing techniques for obtaining the data needed 

c. Reviewing mechanisms for interpreting and reporting the 
data 

d. Reviewing strategies and administrative procedures essen- 
tial in developing and implementing a viable evaluation 
program. 
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General Plan of Operation 



The project involved planning, conducting, and evaluating a one 
week institute held on the University of Arkansas campus, August 4-8, 

1969. The program was planned so as to actively involve the partici- 
pants in a variety of activities designed to facilitate achieving the 
objectives and outcomes previously stated. Emphasis was placed on pre- 
senting and reviewing procedures and techniques of evaluation which are 
effective in obtaining information needed for sound educational decision- 
making. 

Considerable time was devoted to small group sessions where train- 
ees were able to synthesize the presentations made and to develop eval- 
uation guidelines on which a consensus of opinion was reached. Trainee^ 
also developed tentative plans of action describing evaJuation procedures 
and activities which they planned to initiate and/or improve upon return- 
ing to their respective work assignments. 

One hundred participants were selected from among the applicants, 
ninety- two of whom attended and participated in the institute. 
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II. METHODS AND PROCEDURES 



Nomination and Selection of Participants 

Soon after notification that the institute proposal had been 
approved and was to be funded, a letter explaining the purpose and 
objectives of the institute and a nominations form were sent to all of 
the State Directors for Vocational and Technical Education and to all 
of the Directors of State Research Coordination Units. Persons 
receiving the letters were asked to provide the names and addresses of 
up to eight persons from their state whom they felt should be given 
priority consideration as possible participants. Over 250 nominations 
were received, including at least one from every state and territory in 
the nation. 

Additional publicity was given the institute through releases 
which were sent out by the U.S. Office of Education to their regional 
offices, to teacher training institutions, and to the Research 
Coordination Units. 

Counting individual requests and the persons nominated by the 
State Directors and Research Coordination Unit Directors, approximately 
350 persons were sent an application form and brochure (See Appendix A). 
In addition to the nominations form and letter sent in early May, 
three institute brochures, which contained more information about the 
institute program, and application forms were sent to the Directors 
who had not responded to the earlier request for nominations. These 
Directors were asked to forward the materials to persons whom they 
felt might be interested in applying. Brochures and forms were also 
sent to persons who had recently attended two other conferences 
dealing with evaluation. 

The brochure provided interested persons with information on the 
purpose of the institute, a list of the primary objectives, the major 
topics to be considered and the general procedures to be used in 
conducting the institute, A list of the institute staff and 
consultants as well as information on participant selection, travel 
and accommodations was also provided. 



To qualify as participants, applicants had to be state directors 
or supervisors of vocational education, members of state advisory 
councils, assistant superintendents of city schools, directors of city 
vocational education programs, directors of area vocational schools, 
or otherwise responsible for evaluating vocational education programs. 
The following four criteria were the major factors considered in 
selecting the participants: 

1. Present and future evaluation responsibilities. First 
consideration was given to those applicants who had a 
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major responsibility for improving or establishing an 
evaluation program at home base. 

2. Past evaluation activities and experience. An effort 
was made to give special consideration to those who could 
contribute to the institute because of their previous 
experience in evaluation. 

3. The purpose given by the applicant for wanting to attend 
the institute. This information was used in an attempt 

to determine the degree of interest and desire for working 
in evaluation. 

4. Geographic location. Selection was initially made so as 
to include at least one qualified participant, where 

the number and quality of applications allowed it, from 
every state and territory in the country. 



Using the above criteria all applications were carefully 
reviewed by a committee consisting of the institute director, 
associate director, and a representative from the Region VII U.S. 

Office of Education. All qualified applicants from the approximately 
175 applications received were rank ordered on a state-by-state basis 
by each member of the committee. The two highest ranked applicants 
from each state having two or more qualified applicants were selected 
as part of the quota of 100 trainees. Since there were not two 
applicants from every state, additional selections were made from the 
states having a large number of qualified applicants. 

Letters of notification and a pre-registration form were mailed 
along with information on the University of Arkansas and the City of 
Fayetteville to the 100 acceptees. Letters notifying the other 
applicants of their alternate status were also sent. Several of the 
individuals who were initially accepted canceled out prior to the 
start of the institute. As many as possible were replaced with the 
alternates available. 

A list of the ninty-two participants giving their name, professional 
title, and office address is contained in Appendix B. Also included 
in Appendix B is a list of the Institute consultants and supporting 
staff members from the University of Arkansas. 



Planning the Institute 

A program planning committee was organized and convened in late 
May to assist with final plans for the institute. Although the 
institute proposal contained the general objectives, content topics, 
procedures to be used, and a list of potential consultants , considerable 
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work remained. During the one day meeting the planning committee 
helped modify and delimit the original objectives, refine the content, 
sequence the topics, determine time allotments, and finalize he 
daily schedule. Recommendations were also made on the best methods 
for presentation of topics and on how to organize the small group 
dis cussions . 

The planning committee consisted of eight persons including the 
institute director, associate director, a representative from the Ohio 
Center for Vocational and Technical Education, a representative from 
the North Carolina Center for Occupational Education, a representative 
from the U.S. Office of Education, and three other consultants 
representing local, state, and national levels of evaluation. See 
Appendix C for a speciman of the institute program and a list of the 
program planning committee members. 

Another aspect of planning and preparing for the institute 
included the purchase of selected references and the solicitation of 
free evaluation materials. A list of U.S. Office of Education funded 
projects related to evaluation provided one valuable source of 
reference materials. Letters were also sent to the State Directors of 
Vocational Education and the Research Coordination Unit Directors who 
responded to the request for nominees asking them to provide one or 
more complimentary copies of as many useful references on evaluation 
as they had available. Materials were received from approximately 
twenty states as a result of these requests. The Arkansas Research 
Coordination Unit also cooperated by making available all of their 
evaluation related materials including a considerable number of 
microfiche. 



Conducting the Institute 

A wide variety of activities were used to enrich the understandings 
and experiences of the participants. Included were formal lectures, 
informal talks, a symposium, large and small group discussions, 
reaction and questioning panels, small task force assignments and 
reports, and individual assignments. Participants were assigned 
using their preference to one of eight small groups, each of which 
concentrated on one of the following areas of concern to evaluation: 

a. State directed evaluation of statewide programs 

b. State directed evaluation of local programs 

c. State assisted evaluation of local programs 

d. Locally directed evaluation of local programs 

Each consultant prepared a formal paper in advance which was 
duplicated and made available to all participants immediately 
following its presentation. Most consultants were in attendance for 
at least three days and some were present all week. In addition 
to their formal presentations th« consultants were available as 
resource persons to the small groups and for individual consultation. 
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At the close of each day, a short meeting of the institute staff, 
consultants, group leaders and recorders was held to obtain feedback, 
review the next day’s schedule, and to make any changes deemed 
desirable* See Appendix C for a rpeciman of the institute program 
which outlines the specific topics presented and identifies the 
individuals who presented them. 

The facilities of the now Graduate Education building were used 
for all the formal sessions and small group meetings. All of the 
participants and most of the consultants were housed in a modern campus 
dormitory. The two Informal sessions were held in the dining area 
of the dormitory. The reference materials collected and a comprehensive 
list of them were available three evenings of the week in a room 
adjacent to the dormitory lounge. 



Introduction, Welcome, and Orientation 

The institute got underway on Monday morning with registration, 
at which time the trainees were provided with identification tags, a 
list of their fellow trainees, a copy of the institute program, and an 
assortment of materials about the University of Arkansas and the 
Fayetteville area. 

A formal welcome to the University was given by Dr. Henry H. 
Kronenberg , Dean of the College of Education. He spoke briefly 
concerning the importance of evaluating not only vocational but all 
educational programs. 

The institute direcor explained the purpose of the institute, 
reviewed the objectives established for the week long program, and 
discussed the anticipated outcomes. A summary of the various states 
represented and information on the positions held by the participants 
was given. The associate director covered logistical and other 
operational procedures. On the first evening an outdoor social with 
light ref reshments provided an opportunity for trainees and staff to 
become better acquainted. 



Ab stracts of Presentations 



The next portion of this section includes an abstract, prepared 
by the institute staff, of each of the major presentations. 
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ABSTRACTS OF PRESENTATIONS 



THE ABSTRACTS IN THIS REPORT WERE DEVELOPED BY THE PROJECT 
STAFF FROM THE FORMAL PAPERS PRESENTED. FOR A COMPLETE TEXT 
OF EACH PRESENTATION, SEE APPENDIX F. 
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THE ROLE OF EVALUATION IN THE DECISION-MAKING PROCESS 



John K. Coster * 
Robert L. Morgan 



The chain of events initiated by the report of the Panel of Consul- 
tants on Vocational Education in 1963, which led to the enactment of the 
Vocational Education Act of 1963 and culminated in the enactment of the 
Vocational Education Amendments of 1968, has had a profound influence 
on the Office of the State Director of Vocational Education. These 
events have established this position as one of educational statesmanship. 
Not only have the decision-making and managerial aspects of this position 
been increased in geometric proportions, but also the responsibilities 
for changing programs in accord with changing goals has presented a 
difficult task. Not the least of these is program planning and evaluation. 
Inputs must be provided into the decision-making process, and accounta- 
bility for funds dictates that the decision-maker must have access to a 
highly qualified staff, the need for which was not recognized a decade 
ago. 



National goals may be augmented or modified by state and local 
goals. However, acceptance of federal funds which are directed toward 
broad national goals is tantamount to accepting the goals which are 
expressed in the legislative mandates and supporting documents. Objec- 
tives, then, are specified in light of the national goals, modified by 
the state goals. The specification of objectives is the responsibility 
of the decision-maker. Resources are allocated to maximize the attain- 
ment of the objectives. The resource allocation refers to the technology 
of education, that is, to the combination of human resources and hardware 
and software, as well as facilities, which are essential to ♦'he attain- 
ment of objectives. Objectives are assumed to be hierarchial in nature, 
that is, they can be ordered in a hierarchy ranging from the most signi- 
ficant to the least significant objective in light of goals. Outcomes 
are defined in terms of the extent to which the objectives have been 
attained. The evaluation process is directed toward determining the 
degree of congruence between the objectives and the actual outcomes. 

The evaluation constitutes an input into the decision-making processv 
The decision-making process functions in terms of specifying the objec- 
tives and allocating the resources. If the discrepancy between objectives 
and actual outcomes is high, then the resource allocation system must 
be reexamined and decisions made regarding how these resources can be 
reapplied to insure the attainment of the objectives. 

Evaluation must be considered as a high risk activity. The model 
that has been presented, for example, is a conceptual, logical model 



* Dr. Coster is Director of the Center for Occupational Education and 
Mr. Morgan is a Graduate Research Assistant in Psychology at the 
Center for Occupational Education, North Carolina State University at 
Raleigh. 
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which requires considerable work for its implementation. Yet it does 
provide a way of examining the complex of activities which are involved 
in program planning and evaluation; it demonstrates a position of the 
decision-maker and program manager within the model, and it indicates 
in broad terras the information that must be provided by the program 
evaluator to the decision-maker if appropriate alternatives are to be 
selected, objectives attained, and goals realized. The decade of the 
60*s has witnessed a phenomenal advance in educational technology. The 
management of the technology in terras of applying resources to attain 
objectives and realize goals must advance with the technology. The 
head of the program must be a rational man who can make decisions that 
will maximize both the probability of success and the utility of 
attaining objectives. The role of evaluation in the decision-making 
process is to design, direct, analyze, and report the necessary data 
on which decisions may be made. Thus evaluation is not merely essential, 
but absolutely mandatory as a key element in progress and goal 
realization. 




12 



THE SCOPE AND OBJECTIVES OF VOCATIONAL EDUCATION AND THEIR 
RELATIONSHIP TO EVALUATION 

Edwin Crawford* 



The Scope and Objectives of Vocational Education 

The Vocational Education Amendments of 1968 contain a Congressional 
mandate for vocational education to redirect, e>.pand, and reform its 
scope and objectives. 

A. The 1968 Amendments reinforce the Act of 1963 with respect to 
requiring vocational education to redirect its efforts to 
serve the needs of the people as individuals, instead of merely 
providing for the training in certain occupational categories 
to meet manpower requirements. 

B. The increased funding and program authorizations contained in 
the Amendments broaden the areas in which vocational training 
may be offered and increase and specify additional people who 
may be served. 

C. Set-asides for the disadvantaged, the handicapped and post- 
secondary education, plus the separate authorizations for 
special programs, reemphasize major Congressional priorities 
and concerns regarding vocational education. 

D. New and expanded involvement of the private sector in every 
phase of the educational process is another expression of 
Congressional emphasis. 

E. Vocational educators are being asked to develop a planning, 
programming and evaluation system that will ensure that 
Federal funds are being spent in the most effective and effi- 
cient manner possible. Accountability is no longer implied; 
it is now required. 

Relationship of Program Objectives to Evaluation 

A. The main purpose of vocational education evaluation, as is 
the case in any evaluation, is to measure the effectiveness 
and efficiency of vocational education programs, services, and 
activities in terms of prescribed objectives and criteria. 

B. Evaluation is invaluable to vocational educators from a func- 
tional viewpoint, such as assisting them to: 

1. meet legislative requirements, for annual descriptive, 
statistical and financial reports; 



*Mr. Crawford is Senior Program Officer, Program Evaluation, Division of 
Vocational and Technical Education, U.S. Office of Education. 
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2. provide feedback for future program planning and develop- 
ment--^ close the loop in the planning process; 

3. improve management and administrative decision-making; 

4„ determine additional vocational education research needs; 

5. suggest needed legislative changes; 

6. reveal gaps in program coverage and to fulfill the unmet 
needs of students, business and industry, and labor; 

7. discover exemplary programs, services, and activities; 

8. uncover program alternatives for accomplishing prescribed 
objectives; and, 

9. disseminate findings for the edificatior of all concerned 
with vocational education. 

Conclusions 

A. New legislation requires the development of comprehensive plan- 
ning, programming and evaluation systems at the Federal, state, 
and local levels; such requirements should serve as a spring- 
board for program development and improvement. 

B. The prescribed program objectives are stated in terms of target 
groups and target areas to be served, thus reflecting special 
priorities to be emphasized. 

C. The implications of the Congressional mandate for evaluation 
are clear, many and, in some instances, complex. The effec- 
tiveness of current programs and the satisfaction of indivi- 
dual students require that we learn and implement those proce- 
dures and techniques that will most efficiently aid the plan- 
ning and programming process for the attainment of local, state, 
and National objectives. 

D. The various levels of government should share the same concerns 
and priorities regarding planning and evaluation, for their 
cooperative efforts are vital to the development and imple- 
mentation of valid, effective programs of vocational education 
to meet the needs, interests, and abilities of all the Nation's 
youth and adults. 
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A SYSTEM FOR STATE EVALUATION 
OF VOCATIONAL EDUCATION 

Harold Starr* 



State vocational education agencies have used process evaluation 
results to justify the existence of programs, state expenditures to 
local programs, and state agency budgetary requests for personnel and 
financing. One reason that state vocational education agencies may 
require an alternative evaluation strategy is related to the credibility 
of process-oriented results for justifying such budgetary requests. 
Decisions regarding resource allocation by policy-making bodies are being 
based with increasing frequency on evidences of program efficiency, pro- 
grain effectiveness, program relevance to changing social and economic 
conditions, and the degree to which agency programmatic direction re- 
flect community, state, and federal interests and concerns. To this 
end, the demand of policy-making bodies has been for more information 
in terms of programmatic effectiveness and the extent to which state 
agency efforts relate to larger social concerns. Since process evalua- 
tion of vocational program efforts fail to provide evidences related to 
either of these concerns, it cannot be used as a viable strategy for 
conducting state vocational education program evaluation. 

The project staff concluded that an evaluation methodology which 
has greater payoff for state vocational education agency program plan- 
ners in terms of program planning and accountability purposes would oe 
a methodology which is consistant with a systems approach to planning, 
contains process elements, but is oriented primarily toward product or 
outcome measures. Such a systems approach to evaluation methodology 
would require that: (1) the evaluation problem be defined in terms of 

its purposes and expected outcomes; (2) a measurement system would be 
formulated from the types of decision requirements which are logically 
derived from the purposes of the evaluation; (3) proper feedback or 
quality control mechanism would be provided to continously assess the 
effectiveness and efficiency of the information system in providing sig- 
nificant decision-making data; (4) an interpretive system would be for- 
mulated which could analyze and provide information to decision makers 
in a format which would facilitate decision-making; and (5) since the 
evaluation system is only one part of a total program planning system, 
careful attention would be directed to assuring that the evaluation 
system could be articulated with other components of a systematic pro- 
gram planning process. 

The systems approach was designed and tested in two phases: (1) 

May of 1967 to November of 1968, and (2) November of 1968 to September 
of 1969. In September of 1969, the revised approach will be tested in 
four states. The project staff will then work with these cooperating 
states in assessing the program planning procedures which are designed 



*Dr. Starr is Project Director of the Evaluation Systems Project, The 
Center for Vocational and Technical Education, The Ohio State University, 
Columbus, Ohio. 
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to assist the state agencies in developing annual and long-range pro- 
jects of their programmatic activities on the basis of evaluation results. 
Following the field testing of instruments and procedures, a finalized 
evaluation system will be produced at The Center and the materials will 
be disseminated to potential users. 
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STATE DIRECTED EVALUATION OF LOCAL PROGRAMS 
THE PROCESS 

Robert A. Mullen* 



In 1966 the North Carolina Department of Public Instruction ini- 
tiated a state directed evaluation of local programs of vocational 
education. The major purpose of this evaluation was to improve pro- 
grams of vocational education in existence and to insure the highest 
possible quality for programs that were to be implemented in light of 
future available resources. 

The program was set up as a five-year project with 20 percent of 
the local administrative units being evaluated each year. The first 
two years of the project were designed as pilot project years in which 
methods and techniques would be developed and tried out. 

The questions needing answers were: 

1. Are vocational programs in local situations located according 
to occupational opportunities, needs of students and with an 
eye toward comprehensiveness of the school in which it is 
located? 

2. Are programs of vocational education being conducted with 
adequate equipment, materials, including audiovisuals, and- 
are they located in adequate facilities? 

3. Are students receiving adequate guidance and counseling at 
the proper times? 

4. Are students being given assistance where applicable in find- 
ing jobs orce their training program has been completed? 

5. Is instruction timely, up to date, and transferable to 
related occupations or to continuing education opportunities 
and are students receiving a broad outlook toward occupations 
through the application of the cluster concept? 

6. Are local school administrators planning vocational education 
offerings to insure that programs are in keeping with current 
and projected labor market demands? 

7. Are programs within the individual school setting carrying on 
a constant program of evaluation involving student reactions, 
teacher reactions, employer reactions, parent reactions, reac- 
tions of the local school board members and other people 
where applicable? 

8. Is the type of supervision being provided for programs ade- 
quate in terns of program improvement? 

9. Are vocational teachers utilizing a variety of tea^.^ng methods 
and techniques and, in addition, are vocational teachers 
pursuing a program of professional development involving the 



*Mr. Mullen is Associate Director of Program Services, Division of 
Occupational Education, Department of Public Instruction, Raleigh, 
North Cajrolina. 
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improvement of their teaching methods as well as updating and 
upgiading themselves in subject matter areas? 

10. Are adviso y comnuttees being utilized effectively? 

A state staff desiring to initiate an evaluation of local pro- 
grams of vocational education must take into consideration first, 
how it will gain the necessary commitment from the members of the state 
staff to be involved in the evaluation and then, the necessary commit- 
ment from local education agency personnel including superintendents, 
principals of schools, counselors, teachers and coordinators. The 
second consideration or step is to determine the purpose the evaluation 
is to serve. Broadly stated the purposes should be to improve each 
program, with participants at each level, being fully aware of the uses 
to be made of the findings . 

With the above considerations in mind the following 22 objectives 
were determined bv a committee representative of all vocational areas: 

1. To determine the extent to which administration and super- 
vision of vocational education is adequate, both at the 
state and local levels. (General) 

2. To determine the extent and effectiveness of program projec- 
tion, planning, and evaluation at the state level. (General 
and Specific) 

3. To determine the adequacy of state and local involvement in 
local program planning and evaluation for vocational education. 
(General and Specific) 

4. To determine the extent to which vocational education resources 
are allocated according to occupational needs, both useful 

and gainful. (General) 

5. To determine the extent to whicn vocational offerings are 
meeting stated objectives. (Specific) 

6. To determine the level of staff (teachers, coordinators, 
counselors) preparation and the extent co which these indivi- 
duals are engaged in continuous and long-range programs of 
professional development. (General and Specific) 

7. To determine the extent to which vocational teachers plan 
and follow through with sufficient and effective teaching 
practices and experiences suited to student needs. (Specific) 

8. To determine the extent to which appropriate facilities, 
equipment, and teaching materials and supplies are available 
and are used for the various programs. (Specific) 

9. To determine the extent to which vocational teachers contribute 
to career planning of students through program interpretation 
and instructional activities. (Specific) 

10. To determine the extent to which vocational offerings are 
selected by students on a sound career-planning basis. (Specific) 

11. To determine the extent to which teachers and school adminis- 
trators are using experimentation, pilot programs, and innova- 
tions as a means of improving the quality of instruction. 

(General and Specific) 

12. To determine the extent to which vocational education is a co- 
operative effort involving all education. (General and Specific) 
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13. To determine the extent to which students are entering the 
occupational area for which they received vocational training 
and are progressing on the job or continuing training. 
(Specific) 

14. To determine the extent to which the curriculum provit.es for 
students who have special needs. (General and Specific) 

15. To determine the extent to which the needs of adult farmers 
and homemakers are being met. (Specific) 

16. To determine the extent to which vocational teachers are 
involved in school activities. (Specific) 

17. To determine the extent to which advisory committees are 
used in planning new programs and improving existing programs. 
(Specific) 

18. To determine the extent to which community resources are used 
in the various instructional programs of vocational education. 

19. To determine the extent to which vocational teachers, coun- 
selors, coordinators, and administrators interpret the voca- 
tional education program. (General and Specific) 

20. To determine tne extent to which youth organizations strength- 
en and extend the various vocational programs. (Specific) 

21. To determine the extent to which vocational teachers assume 
professional leadership beyond the instructional program. 
(Specific) 

22. To determine the extent to which vocational offerings are 
available to students on a fee-free basis. (Specific) 

It may be noted that some objectives are both general and speci- 
fic, while others are only general or specific. General means it is 
applicable to all vocational education, while Specific applies to each 
subject area. 

These objectives were treated within the framework of the follow- 
ing four basic questions: 

1. What is the preseat status of vocational education in the 
State of North Carolina in each of the classrooms, shops and 
laboratories? 

2. What ought vocational education be in the coming year? 

3. Wiiat are the problem areas that exist in vocational education 
in local programs? 

4. What will be the future course of action for local programs? 

Experience in this evaluative process has indicated several 
desirable points to be considered for the future of the process. Among 
these are; the desirability of involving local personnel in inservice 
programs so that they might become more intimately involved in the eval- 
uation process, gradually shifting a portion of the responsibility from 
the state level to the local level; that a reorganization at the state 
staff level take place so as to provide increased general supervision 
in addition to supervision by specialized areas; and, that increased 
emphasis be put upon working with personnel at the local level so that 
the result of future evaluations may play a more important role in the 
future of programs at the local level. Experience has also shown that 
the results of the evaluation are important in developing long-range 
plans for the future of vocational education at the local level. 
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PROJECT IMPROVE; 

THE DEVELOPMENT OF A STATE OPERATED EVALUATION SYSTEM 

Jerome Moss, Jr., Brandon B. Smith, 

Frank Pratzner, David Wheeler, 
and William Stock * 



The title of this paper is slightly misleading. The Minnesota 
Research Coordinating Unit is not developing a complete system for 
evaluating vocational programs in the State; it is developing only a 
subsystem at present for studying post -secondary level programs. The 
kinds of questions this subsystem will seek answers for are micro- 
level, qualitative and diagnostic in nature. How "good", how 
"successful", is the automechanic curriculum in school X? How effi- 
ciently is it operating? What can be done to improve its effectiveness 
with minimum increase in costs? These are the kinds of questions the 
RCU is attempting to answer. 

The subsystem will provide educational managers and program devel- 
opers at the State and local levels with information useful in making 
investment decisions about, and in improving the quality of, specific 
vocational programs throughout the State. When completed, the sub- 
system should be readily integrated with a total management informa- 
tion system and be adaptable to all levels of vocational instruction. 

The project is planned in three phases* (a) Phase 1 will design 
the overall subsystem and identify relevant variables; (b) Phase II 
will de/elop the instrumentation and the data collection and process- 
ing procedures; (c) Phase III will tryout and revise the subsystem. 
Phase I is now nearly complete. 

Before attempting to design the subsystem, certain issues in 
evaluation were examined and assumptions made about them which served 
to create the framework within which subsystem development could pro- 
ceed. 



First, the subsystem should’ employ the "products" of specific 
vocational programs --former student-learner behavior — as evaluative 
criteria. 

Second, diagnosis of instructional programs--the ability to deter- 
mine how effectiveness can be improved--is an essential aspect of a 
subsystem directed toward assuring program quality. 

Third, vocational program management decisions cannot be made 
without regard to costs. 



* Dr. Moss and his colleagues all are members of the Minnesota Research 
Coordination Unit for Vocational Education at the University of 
Minnesota. 
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Fourth, all evaluation requires comparison with some standard. 

This evaluation will determine the relative effectiveness and efficiency 
of two or more "equivalent" vocational programs. 

Fifth, to make possible maximum efficiency, objectivity, and useful 
comparability, a subsystem designed to provide periodic program evalua- 
tion should be state-operated. 

In order to develop the overall design of the evaluation subsystem 
within the assumed prerequisites, consideration was given to five basic 
questions, the first three of which are discussed in some detail. 

How should vocational programs be classified so that we can make 
logical, useful comparisons between Lhem, as well as provide the basis 
for future generalizations of results? For purposes of evaluation only 
programs of similar characteristics with similar students may be com- 
pared fairly. 

Since purposes dictate desired outcomes and, in turn, criterion 
measures, what is to be assumed about the role and purposes of post- 
secondary vocational education? It is assumed that criterion measures, 
in terms of former student learner behaviors will be based upon program 
purposes, that is, the principal purpose of vocational education is to 
improve the work adjustment of the individual as measured both by worker 
satisfactoriness and satisfaction. — — 

What are the educational and other factors that influence educa- 
tional outcomes? That is, what is the educational production function? 
The general form of the production function is: 

0 = f [p, C] 

0 = Outcome measures of the program 

f = Function of program characteristics (P) 
and constraint factors (C) 

P = Program characteristic (manipulable by program 
developers and operators) 

C = Constraint factors (not manipulable by program 
developers and operators) 

For further elaboration upon this general production function, 
the reader is referred to the complete text contained in Appendix F. 

What kind of cost function should be developed? 

What decision criteria should be employed to determine the program 
in which additional funds should be invested? 

When data on outcomes, program characteristics, and constraint 
factors become available for two or more programs, the production func- 
tion can be treated as an equation susceptible to regressional analysis. 

It is recognized that the relationship revealed by the type of 
analysis envisioned are not necessarily causal. But given the logic of 
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the production functions, the relationships found do provide a reason- 
able basis for assuming a degree of causality, especially when the pre- 
dictive value of the independent variable in the production function 
proves to be high. 

The Minnesota Research Coordinating Unit is in the process of 
designing an evaluation subsystem which can be integrated with more 
complete management information systems, and which will yield evalua- 
tive and diagnostic information about the relative efficiency and 
effectiveness of individual program offerings. 
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LOCALLY DIRECTED EVALUATION OF LOCAL PROGRAMS 
OF VOCATIONAL EDUCATION 

Harold M. Byram * 



There are three general strategies for bringing about evaluations 
of local programs of vocational education. One strategy may be called 
a state initi ated and/or state operated evaluation. In this type evalu- 
ation” the state educational agency is generally responsible for deter- 
mining objectives, developing criteria or criterion questions, and 
deciding on the scope and depth jf data needed to answer them. The 
instruments are state developed and uniform for all districts. 

A second strategy may be referred to as a state led evaluation of 
local programs. Using this approach the state agency or a university 
provides a program to prepare local administrators and teachers who 
conduct their own evaluations with the aid of staff committees and 
citizen advisory committees. The local leaders are given help in des- 
cribing their goals and objectives, in formulating criterion questions, 
in developing or selecting instruments for gathering relevant data, 
in analyzing and interpreting the data, and in effectively involving 
both staff and citizens in the whole process. 

The third strategy is independent local program evaluation. Here, 
a local district would, without previous training or consultant help, 
conduct its own evaluation. It would not necessarily report to any 
entity outside the district and might or might not conduct the study 
on as broad a scope as in the case of the other two strategies. 

Each of the strategies has certain merits and shortcomings. A 
major advantage of the state led local evaluation is that it provides 
for a process in which those who are responsible for, and those affec- 
ted by the program conduct the appraisal for themselves. It also calls 
for the training of local educators in the processes and procedures of 
evaluation by either the state agency or state universities. 

Most of this presentation deals with state led local evaluations 
because of a strong bias toward local involvement and program planning 
and because of six years' experience with this strategy. 

Several essential elements of local program evaluations are 
identified as follows: 

(1) There must be a commitment by the local administration that 
goes beyond mere sanction or endorsement. Those who are 
involved in conducting a local evaluation need to be assured 
that they have support for their activities, that resources 



* Dr. Byram is Professor of Education, Department of Secondary Educa- 
tion and Curriculum, College of Education, Michigan State University. 
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for completing them will be provided 3 and that serious 
consideration will be given to the findings a».d recommenda- 
tions . 

(2) A competed, strategically placed local leader is also 
essential. A director or supervisor of vocational education 
or similar person who has the recognition and respect of all 
administrators and other professional staff members is needed. 

(3) The evaluation leader or chairman, and the staff committee 
working with him should be trained in research and evaluation 
procedures. These persons will need to select, modify, or 
develop instruments for retrieving data, to monitor the data 
processing, and to analyze the data collected. 

(4) Another essential element is a functioning advisory committee 
or committees. Representatives of people affected by the 
program should have a voice in evaluating it. Furthermore, 
they represent a source of valuable information that is basic 
to local program planning and are more likely to accept and 
help implement recommendations if they have been actively 
involved. There may be one general advisory committee, a 
committee for each occupational area, or a combination of 
these. 

(5) Because staff members will make a significant input in terms 
of energy and time, it is essential that they be provided 
time to work on the activities planned. Adequate clerical 
services need to be made available too. 

(6) Consultant services will, in many cases, also be essential. 
Consultants should have an understanding of modem principles 
and programs of occupational education and have the special- 
ized competencies that the local educators want. 

Beyond the elements which have been categorized as essential, 
there are additional ones which would be designated as desirable by 
most people experienced in local program appraisals. The focus of the 
local program evaluation needs to be on outcomes. This is not to say 
that the input elements can be ignored, but until there can be an 
appraisal of the output of a program, there is really little basis for 
considering change in ' • input. The major question to be answered is: 
Are the results of the program commensurate with the expectations held 
by those responsible for and affected by the program? 

Another desirable element is that the scope of the evaluation be 
broad. It should not be limited to programs that are reimbursable, nor 
to programs that have been designated as vocational. Of course, many 
programs will contribute more directly to occupational preparation than 
will others. 

This system, if it can be called that, has worked in the states of 
Michigan, Arkansas, Minnesota, Mississippi, and Nevada where it has been 
tried in both area and local schools, and in schools varying in size 
from 2,500 to 30,000 students (K-12). 
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The urgency of improving local programs of vocational and techni- 
cal education through evaluation and program planning is too great to 
bo left to chance. If the opportunity which local educators and citi- 
zens now have is not taken advantage of, there is a strong possibility 
that it may pass out of their hands. 




25 



VIEWPOINTS OF A MEMBER OF THE NATIONAL ADVISORY COUNCIL 
FOR VOCATIONAL EDUCATION 

Jack Michie * 



A careful look at the teacher education programs and credentialing 
systems within the states would be a great starting point for an evalu- 
ation. Too often we avoid the real question as we attempt to evaluate: 
"What are we doing for the kids?" That's very simply stated and very 
real. That's where the whole problem lies. 

What to evaluate? Evaluate the scope and the goals of vocational 
education, the organization and administration of vocational education 
and we certainly have to look at the program results and what happens . 
What happens to people after they complete a program? One day Congress 
is going to ask how we spent our money and what we did for people. 

Whether a new area of vocational education is being considered cr 
an exiting program is being evaluated, administrator::; must heed the 
expectations of . e community. Three very salient questions should be 
asked. What are the expectations of business and industry in the 
community? What are the expectations of the advisory councils and 
committees? What are the expectations of the students? 

The National Advisory Council is committed in these ways: 

1. Seeing to it that career planning is provided for each 
citizen. 

2. Assisting persons to develop an interest in vocations 
and pursue occupational exploration. 

3. Allowing the student to participate in varied work 
experience so he can identify and accept for himself 
the dignity of occupational choice. 

4. The strengthening of occupational choice and evaluating 
the relevance of the total occupational process. 

5. The initiation, encouragement, and support of cooperative 
effort among .*11 elements r components of the educational 
system in the nation. 

6. The elimination of artificial barriers to success in 
personnel development, and encouragement of flexibility 
in facilitating and capitalizing upon the differences and 
strengths which individuals possess. 

7. The encouragement and support for creative und innovative 
programs, and a continuous pursuit of financial support 
for vocational education, and a development of a national 
plan for vocational education to prepare people for work, 
and a monitoring system which will provide for effective 
evaluation. 



* Dr. Michie is a member of the National Advisory Council for Vocational 
Education and at the time of this presentation was Dean of Institutional 
Planning, Southwestern College, Chula Vista, California. 
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If education is to be relevant, if schools are to be real, if our 
obligations to young people are to be fulfilled, and if they are to be 
given a future better than the past, then we need to reconstitute our 
total educational system so that vocational education becomes education. 
There is no other kind. 
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FOLLOW-UP PROCEDURES FOR VOCATIONAL EDUCATION STUDENTS 



Richard WTiinfield * 



The follow-up is merely a technique for gathering data and must 
be properly related to the grand design for any evaluation project. 

M A follow-up study is an attempt to accumulate relevant data from or 
about a group (or groups) of individuals after they have had certain 
similar or comparable experiences or who have certain similar or 
comparable characteristics Although the usual conception of a 
follow-up implies the use of a mailed questionnaire, there are many 
kinds of data about people which can be collected by other means. 

The decision to conduct a follow-up should result from an analysis 
of the problem to be solved or the questions to be answered. If the 
best way to get the data needed through use of the follow-up tech- 
nique, then it should be employed. 

Follow-ups may serve one or more of three major purposes depend- 
ing on the design and intent of the investigator. The first is to 
obtain descriptive data about what has happened to people, which may 
be of some usefulness in predicting with limited accuracy, what may 
happen to subsequent similar groups. This approach has limited evalua- 
tion potential for it requires arbitrary judgment to conclude whether 
the description is good or bad. 

A s cond purpose of a follow-up study may be to elicit evaluative 
responses of former students about their educational experiences. The 
problem here is that the respondees have a limited knowledge of alterna- 
tives, and each respondee speaks out of a different frame of reference. 

As an evaluation or advisory device this approach has the most useful- 
ness when the occasional consensus is found. 

The most useful purpose of a follow-up is a comparative one. 

There are numerous comparisons which can be made between groups such 
as dropouts, academic students, students from different schools, students 
from other states, and students taking other courses. 

There are four major considerations or components of any follow- 
up study. The first concern deals with the persons to be included in 
the study. An accurate definition of the group or groups to be studied 
is essential if the study is to be valid. A decision must also be 
made after considering time, cost, and other factors whether to study 
the whole universe or to select a sample. 



* Dr. Whinfield is Administrative Associate, Center for Studies in 
Vocational and Technical Education, The University of Wisconsin, 
Madison. 
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A second major component is the data gathering instrument itself. 

If the v./rong data is obtained, the study is of little value. The 
development of a good questionnaire is a very difficult process unless 
you are seeking only minimal information. Forced response versus 
open-ended questions is one of many considerations needing careful 
attention when developing the instrument. Forced responses are easier 
for respondees to handle and lend themselves to easy statistical treat- 
ment, but they are dangerous in that they may not provide for all the 
alternative answers. The design or format of the questionnaire, includ- 
ing size and legibility are factors influencing the response rate. It 
is recommended that the instrument be pre-tested to determine whether 
or not the questions are understood, to improve the response rate, and 
to increase the reliability of the questionnaire. 

The third factor of primary concern is the response rate. The 
response rate is directly related to the sources of the request for 
completing the questionnaire. The wording, length, and design of the 
request letter is also of significance. Confidentialness should be 
assured and an honest, straight-forward appeal made. A rapid mailing 
sequence of approximately seven day intervals appears to give better 
results than mailing with intervals of two or three weeks. 

The fourth major component of concern in conducting a follow-up 
involves data handling and treatment. A variety of techniques are 
used in coding. The actual coding of questionnaires frequently 
requires some specialization and close supervision. The goal to 
strive for is consistency in handling each problem. The general rule 
is to code everything in raw form and then transform and categorize 
it by machine. Treatment of data should be determined in a general 
way before you start collecting data. 

In conclusion, two general statements: 

1. There is no substitute for good data. While it is 
perfectly possible to treat the accumulation of data 
casually, without any reader of the final report being 
aware of the shortcoming, the integrity of a researcher 
demands that he do all in his power to get the best data 
possible . 

2. To assume this is an easy process is a grievous error. 

The attention to careful planning, to thoughtful and 
conscientious control will eliminate many frustrations 
which come from treating and reporting data. 
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ARKANSAS VOCATIONAL STUDENT INFORMATION 
AND FOLLOW-UP SYSTEM 

William C. Amwine * 



The purpose of the system is to collect relevant data on students 
as they enter educational institutions, as they exit, and periodically 
after they have left school. The data gathering system is designed to 
provide school administrators with the information they need for making 
sound decisions regarding program improvement and program expansion. 
Analyses of the data will, among other things, serve to measure the 
effectiveness of instruction, curriculum, and student personnel services. 

The Vocational Student Information and Follow-Up System designed 
specifically for post-secondary programs, is concerned with how well 
the students are being prepared for employment. It provides for the 
collection of data needed to answer several important questions: 

1. Are the students receiving an education which permits 
them to start work with competence and assurance? 

2. Are the students well received by their employers as 
evidenced by a higher rate of pay and rapid promotion? 

3. What are the students* opinions and attitudes toward the 
type of training they received? 

4. Is the training program supplying adequate numbers of 
skilled workers for the businesses and industry in the 
community? 

To be effective, an effort must be made to involve and inform 
all staff, as well as students, about the system and its potential 
for program improvement. Proper conditioning and positive attitudes 
on the part of the students are particularly important as they will 
affect the students cooperation in accurately completing forms while 
they are in school and in returning future mailed questionnaires. 

The information and follow-up system is composed of three impor- 
tant phases: Phase I is concerned with the collection of data; Phase 

II with the analysis of the data; and Phase III with decision-making 
and converting the information into action. 

Phase I, data collection, involves the use of three forms. The 
first form or enrollment form is completed about two weeks after the 
student enters school and identifies the student with respect to name, 
age, sex, and the type of training desired. The second form or exit 
form is completed by students about a week before graduation or 
whenever they exit for any other reason. This form provides for up- 
dating some of the data gathered on the entry form, provides informa- 
tion on program completion and job status, as well as providing for 



* Dr. Amwine was employed as a Project Consultant by the Arkansas 
Research Coordination Unit for Occupational Education, Department of 
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some evaluative responses concerning several factors related to the 
student f s instructional program. The third form or follow-up form 
is partially completed by computer, which fills in the school name and 
the student* s name and address as he entered it on his exit form, 
before being mailed to the former student. After receiving it, the 
former student answers pertinent questions concerning his current status 
and training program. The follow-up form is pre-addressed and stamped 
so that it may be easily folded, stapled, and mailed to the Vocational 
Research Center within the suggested ten-day period. 

Phase II, data analysis, is computerized to facilitate data hand- 
ling and interpretation into meaningful reports. Several standard 
reports will be generated not only for the purpose of measuring the 
effectiveness of present-day instruction, curriculum, and student 
personnel services, but also to lay a foundation for planning future 
programs to better meet the needs of students and the community A 
variety of special reports could also be produced to provide greater 
insight into specific programs or problem areas. The reports can be 
as detailed as needed or in summary form. They can be run to satisfy 
the data needs of administrators at all levels and the data needs of 
instructors in optimizing their effectiveness and efficiency relative 
to their job responsibilities and institutional objectives. 

p hase III or the action phase, is a crucial one in terms of 
improving vocational-technical education programs. Administrators and 
teachers are key personnel in this phase as only they can implement the 
recommendations that result from this information system. Implementation 
should take the form of increased attention to constantly reviewing and 
updating curricular offerings and course content to insure the type of 
training demanded by business and industry and by our ever-changing 
technological society. 

The system provides a medium through which educational programs 
can become more efficient as well as more responsive to both student 
and manpower needs . 
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VIEWS FROM THE CHAIRMAN OF A STATE W "ATIONAL 
ADVISORY COUNCIL 



Caroline E. Hughes * 



State advisory councils for vocational and technical education are 
reasonably new, resulting from the legislative Amendments of 1968, where 
their formation, support and duties were clearly defined. These councils 
are formed by a group of previously unacquainted people, with varied 
backgrounds of experience, but mutual desire in education, social and 
economic progress-, The diversified background of membership provides a 
knowledgable and foiled able resource for the evaluation of our 
educational process. 

Advisory councils are organized to be working bodies, with authority 
for research and recommendation to state departments with whom the 
responsibility for implementation rests. 

Oklahoma Council members have talked informally and frequently regarding 
personnel requirements for efficient and expedient Council operation, 
and attempted to define specific directions which an advisory group may 
prudently take. A general philosophy is evolving. Staff personnel is 
being thoughtfully considered, pending the crystallization of Council 
direction and budgetary appropriations. 

The Advisory Council is cognizant of an extensive and growing 
vocational and technical program under the excellent professional guidance 
of the State Department. It seeks to cooperate with the Department in an 
expanding program, yet maintain an objectivity to insure maximum 
advisory effectiveness. 

The Council shares the views of a representative for Western Electric 
in Oklahoma City who gave several suggestions for improving the present 
educational system in order to prepare students for entry into the world 
of work. His suggestions included: 

1. Prepare the children so that they can at least read a 
newspaper and organize sequences of numbers. 

2. Develop in the children positive attitudes toward work. 

People who don't want to work, won't show up for work, 
won't notify employers, do not possess a sense of 
respons ib i 1 i ty . 

3. Schools must stop promoting the individual and promote 
group activities. In business and industry one must work 
for the good of the group and not the individual. 

4. Information received on personal recommendations is most 
unreliable. Recommendations from professional people tend 
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to be the most unrealistic of ail. Students have a very 
difficult time in applying for a job. They don’t know how. 
No one has ever bothered to inform the student where and 
how to look for a job or how to apply for a job. The 
majority of the educators consider that all the graduating 
seniors will enter college, therefore,, their prime concern 
is how to select a college. 

Though the responsibilities of advisory councils are statewide, 
they share in local and national concerns. They represent a valuable 
resource for the state departments of vocational education. 
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COST-EFFECTIVENESS ANALYSIS AS A METHOD FOR THE 
EVALUATION OF VOCATIONAL AND TECHNICAL EDUCATION 



Jacob J. Kaufman * 



It is the purpose of this paper to discuss cost-benefit analysis 
in terms of (1) its logic and meaning; (2) some of the misconceptions 
which prevail concerning this method of evaluation; (3) some of the 
problems and limitations of this method; and (4) the conclusions of 
a study, conducted by the Institute for Research on Human Resources at 
The Pennsylvania State University, which attempted to determine 
whether or not there is pay-off from an investment in vocational and 
technical education. 

The fact is that there is a tendency on the part of some educators 
to talk simply in terras of the ''needs" of education. Their position 
is simple: the governmental agency should raise whatever funds are 

necessary to meet these "needs". On the other hand, there are some 
politicians who assert that there is a fixed sum of money available 
for educators to spend on education. The fact is that one should not 
talk about education in terms of cost or needs alone. No cost can be 
justified without a reference to pay-off. And the satisfaction of any 
need cannot be justified without reference to cost. 

But what control do we have over public education? What incentives 
are there for the public educator to keep his cost down? What 
evidence is there that public education is being provided efficiently? 
What evidence is there that the objectives are being achieved? It 
is being suggested that these are legitimate questions to ask and that 
there appears to be only one appropriate method for arriving at the 
answer, that of cosr.-benef it analysis. 

One aspect of cost-benefit analysis which should be stressed is 
that it is basically a "way of thinking." It tends to force an 
administrator to think through his objectives. Too frequently they 
are expressed too broadly and do not reflect the "real" objectives. 
Second, it forces an administrator to concentrate on costs as well as 
objectives. Third, it forces an administrator to think in terms of 
alternatives . 

The six misconceptions of cost-benefit analysis are: (1) it is 
merely a subterfuge for seeking to conduct education on a "least cost" 
basis, (2) benefit is measured only in dollar terms, (3) there are 
some things that are not quantifiable, (4) the cost-benefit technique 
has not been fully developed, (5) there is a misconception that the 



* Dr. Kaufman is Professor of Economics and Director, Institute for 
Human Resources, The Pennsylvania State University. 



cost-benefit analyst substitutes his judgment for that of the decision- 
maker, and (6) it is sometimes argued that cost-benefit analysis tends 
to ignore political considerations. 

A major distinction must be aade between evaluation of individuals 
and evaluation of processes. Most educators stil? tend to think of 
evaluation only in terms of testing, or :Ln terms of discriminating among 
individual students for administrative or instructional purposes. 

The goal is not the assessment of the individual but rather the 
assessment of the progress of all students within a program and the 
determination of reasons for all the relative success of various 
aspects of this program. 

The definitional problem centers around a distinction between 
measurement and evaluation. The distinction between the two is 
important. Measurement implies only quantity, while evaluation implies 
quantity plus quality. Measurement is a necessary part of evaluation, 
but evaluation requires both pre-measurement and post-measurement 
considerations. Before measurement commences, evaluation requires 
the formulation of a basic educational philosophy (and its attendant 
goals) and the statement of specific behavioral objectives to be 
measured. After measurement is completed, evaluation requires 
(1) the analysis of measured quantities in terms of the attainment of 
objectives and progress toward goals, (2) an estimate of the value of 
existing programs in determining this progress and (3) an estimate of 
the costs involved in conducting these programs. 
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Evaluation Procedures 



The evaluation of the institute was planned in an effort to 
determine the degree of success, or failure of the institute. Both 
objective measures and subjective measures were used. 

At the close of the institute, participant satisfaction with the 
total training program was assessed by using a Likert-type instrument 
adapted from a form developed by J. Eugene Weldon. A copy of the form 
used is located in Appendix D. 

Another aspect of the evaluation scheme involved having each 
participant or group of participants develop a project relevant 
to his or their area of responsibility. A summary of the types of 
plans developed is provided in the next chapter of this report. 

The third measure of the effectiveness of the institute was a 
follow-up form which was sent to all participants approximately 
eight months after the close of the institute. Participants were asked 
to indicate what new evaluation activities they have initiated and 
what activities they have modified. Questions were also asked to 
determine the number of personnel trained by persons attending the 
institute and to determine the extent to which plans of action 
developed were carried out. See Appendix D fcr a copy of the follow- 
up instrument. 
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III. EVALUATION 



The evaluation scheme consisted of three types of assessments. 

They include: (1) a measure of participant satisfaction with the total 
institute program, (2) a summary of the immediate outcomes of the 
institute in terms of the plans of action which were developed during 
the week of the institute, and (3) a follow-up survey of all participants 
approximately eight months after the institute to determine the 
evaluation activities of the participants since the institute. These 
procedures are discussed in greater detail in the remainder of this 
chapter and copies of the two forms used are presented in Appendix D. 

While most of the evidence presented was gathered specifically for 
the purpose of evaluating the institute, the authors would like to take 
this opportunity to report that there were many favorable reactions from 
the trainees during the institute and in several letters received by the 
staff since the institute. 

Satisfaction With Total Program 

At the close of the institute, each participant was asked to 
anonymously register his degree of satisfaction with the total 
institute program by completing an Institute Evaluation Scale consist- 
ing of two parts. Of the 92 participants, 79 submitted evaluation forms. 
Part I consisted of thirty-five statements about the institute, and 
Part II consisted of two questions and two statements asking partici- 
pants to indicate their feeling about various aspects of the program. 

The thirty-five statements, of which eighteen were couched in negative 
terms and seventeen in a positive format, were reacted to using a 
five-point Likert type scale. These statements and a summary of the 
participants 1 responses to them are presented in Table I. 

Rather than present the data derived from the rating scale in 
detail the reader is encouraged tc study the distribution of responses 
as well as the mean for each of the statements. The following 
discussion therefore includes only a few of the highlights. 

The participants showed agreement with all the positive statements 
and disagreement with all but one of the negative statements. The 
reaction to statement 14, "The information presented was too theoreti- 
cal," although positive was only slightly so. The mean values 
indicate that participants agreed most with statements 1, 7, 15, 16, 

23, 24, 30, and 32; and disagreed most with statements 6, 10, 22, and 
31. The general tendency for participants to agree with the positive 
evaluative statements and their tendency with one exception to disagree 
with the negative statements, suggests a fairly high degree of satisfac- 
tion with the total program. 

The first question in Part II of the Institute Evaluation Scale 
asked the participants to indicate the two most important ways they 
planned to apply the outcomes obtained from attending the institute. 

A summary of the responses to this question is presented in Table II. 
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TABLE I 



MEAN VALUES AND FREQUENCY DISTRIBUTION FOR 
PARTICIPANTS 1 EVALUATION OF THE INSTITUTE 



Frequency Distribution 







G Strongly 
Agree 


<u 

u 

00 

< 

(4) 


G Undecided 


^ Disagree 

V-/ 


^ Strongly 
w Disagree 


1 

Mean 


1. 


Statements 

The objectives of this 
institute were clear to me . . . 


17 


48 


5 


6 


3 


3.89 


2. 


Specific purposes made it ea^y to 
work efficiently 


3 


45 


11 


16 


4 


3.34 


3. 


The participants accepted the 
purposes of this program .... 


7 


46 


12 


8 


6 


3.51 


4. 


The purposes of this institute 
were not realistic 


6 


13 


3 


42 


13 


2.41 


5 . 


The objectives of this program 
were not the same as my objectives 3 


23 


6 


36 


11 


2.63 


6 • 


I didn’t learn anything new . . 


2 


5 


2 


35 


35 


1.78 


7 . 


The material presented was 
valuable to me 


18 


45 


7 


5 


4 


3.86 


8. 


I could have learned as much by 
reading a book 


4 


5 


4 


43 


23 


2.04 


9. 


Possible solutions to my 
l roblems were considered .... 


8 


46 


10 


10 


5 


3.53 


10. 


The information presented was 
too elemental y 


0 


5 


7 


44 


23 


1.92 


11. 


The speakers really knew their 
subjects 


8 


48 


13 


8 


2 


3.66 


12. 


The discussion leaders were not 
well prepared . 


12 


16 


6 


34 


11 


2.80 
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TABLE J (cont) 



Frequency Distribution 
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u 
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00 
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D 
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0) 


0) 


fH 


0) 


M 


00 


M 


00 


a 


00 


P3 


o 


03 
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M 


W 


•H 


+-> 


*H 


P 


CO 


P 



(5) (*) (3) (2) (1) 



e 

n) 



13. 


I was stimulated to think object- 
ively about the topics presented 


7 


53 


10 


9 


0 


3.73 


14. 


The information presented was 
too theoretical 


14 


26 


6 


28 


5 


3.20 


15. 


New acquaintances were made 
which will help in my future 
work 


37 


40 


2 


0 


0 


4.44 


16. 


We worked well together as a 
group 


25 


40 


7 


7 


0 


4.05 


17. 


We did not relate theory to 
practice 


2 


19 


13 


39 


6 


2.65 


18. 


The sessions followed a 
logical pattern 


12 


49 


4 


13 


1 


3.73 


19. 


The schedule was too fixed. . 


11 


20 


6 


40 


2 


2.97 


20. 


The group discussions were 
excellent 


11 


42 


13 


12 


1 


3.63 


21. 


There was very little time for 
informal conversation. . . . 


7 


15 


8 


39 


10 


2.54 


22. 


I did not have an opportunity 
to express my ideas .... 


0 


5 


5 


52 


17 


1.97 


23. 


My time was well spent. . . 


15 


46 


13 


1 


4 


3.85 


24. 


I really felt a part of this 
group 


10 


58 


9 


1 


1 


3.95 


25. 


The program met my expectations 


8 


45 


11 


11 


11 


3.59 
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TABLE I (cont) 



Frequency Distribution 







p Strongly 
Agree 


0) 

v 

M 

60 

< 

14 X 


p Undecided 

LO 


QJ 

CD 

M 

60 

03 

CO 

•H 

Q 

(2) 


^ Strongly 
Disagree 


Keans 


26. 


I have no guide for future 
action 


1 


5 


5 


50 


18 


2.00 


27. 


Too much time was demoted to 
trivial matters . . , . . 


1 


8 


12 


43 


15 


2.20 


28. 


The information presented was 
too advanced. „ 


3 


14 


5 


44 


13 


2.37 


29. 


The content presented was not 
applicable to occupational 
programs 


3 


8 


9 


46 


13 


2.27 


30. 


Institutes of this nature should 
be offered again in future years 


32 


29 


13 


3 


2 


4.09 


31. 


Institutes such as this will 
contribute little to vocational 
education 


4 


2 


7 


33 


33 


1.87 


32. 


The information presented will 
be useful to me as a 
vocational educator 


24 


43 


9 


2 


1 


4.10 


33. 


The references available to 
participants were not appropriate 1 


5 


13 


45 


15 


2.14 


34. 


The facilities for the institute 
were inadequate 


2 


13 


7 


36 


21 


2.23 


35. 


The d oral Cor> and foou service 
was excellent 


19 


30 


4 


20 


6 


3.46 
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TABLE II 



SUMMARY OF RESPONSES OF FIFTY-SIX PARTICIPANTS TO 
QUESTION ON THE TWO MOST IMPORTANT WAYS THEY 
PLANNED TO APPLY THE OUTCOMES OBTAINED FROM THE 
INSTITUTE * 



Outcome 


No. of 
Responses 


% of Total 
Responses 


Establishing an evaluation program 
for my state 


22 


19.6 


Increased ability to advise others 

in planning occupational programs 


20 


17.9 


Establishing an evaluation program for my 
school or institution 


15 


13.4 


Planning of vocational programs 


14 


12.5 


Improvement of teaching or supervision 


10 


8.9 


Administration of occupational 
programs 


9 


8.0 


Use in present research studies 


8 


7.1 


Writing an article or other publication 
on this topic 


2 


1.8 


Increased knowledge of research in 
general 


2 


1.8 


Preparation of curriculum materials 


2 


1.8 


Preparation of a research project 


2 


1.8 


Other 


6 


5.4 



* Twenty-three participants either failed to respond to this question 
or their responses were unusable. 
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Participant responses shown in Table II indicate they were expecting 
to use the knowledges gained in a wide variety of ways. The major 
outcomes planned were to establish an evaluation program for their state, 
school, or other institution, and to become involved in either the 
planning of vocational programs or advising others on the planning of 
occupational programs. The establishment of an evaluation plan or 
scheme for use at home base was one of the planned outcomes of the 
Institute. It is logical that such evaluation programs would in turn 
be used to improve vocational programs or to advise others on the 
improvement of such programs. 

The second item in Part II asked the participants to indicate 
areas or topics which they felt should be included in future institutes 
on evaluation. Most of the topics listed had been covered to some 
extent during the week but responses to this item indicates somn were 
not covered in sufficient depth. Nine topics which were suggested by 
three or more persons are listed below in descending order of their 
frequency : 

Review more completed research projects which have utilized 
a variety of proven practical evaluation techniques 
More attention to preparing and using measurable behavioral 
objectives 

More on the development and selection of evaluation instruments 
More on cost-benefit (effectiveness) procedures 
Time to prepare a step-by-step approach to evaluation 
More on interpreting data and formulating recommendations 
Evaluation of specific programs, i.e., disadvantaged, counseling, 
etc. 

Techniques for dissemination and implementation of findings 
Mc^e practical application of topics presented 

The third item in Part II requested the trainees to write out 
any "suggestions which would provide direction for planning and 
improving future institutes." Due to the open-ended nature of the 
question, a large variety of responses, some of which conflicted with 
each other, were obtained. The suggestions offered by three or more 
persons, with the most frequent listed first, were a n follows: 

More time should be devoted to small group work 
More detailed instructions and training should be given 
small group leaders and recorders 
Arrange for more recreational opportunities 
Don’t schedule the program so tightly 

Visit a model vocational program to serve as a framework for 
relating all presentations 

Provide more opportunity for dialogue between presenters and 
participants 

Plan more organized activities for dependents 
Very well planned and executed 




42 



The fourth and final item in Part II of the Institute Evaluation 
Scale asked, ”What comments, positive or negative, can you make which 
will assist us in evaluating the institute?” Again many comments both 
positive and negative were received. Seven participants interpreted 
the question as asking what procedure or procedures could be used to 
further evaluate the institute. All seven suggested a procedure 
which nad already been planned - that of conducting a follow-up within 
one year to see if the participants have modified or initiated new 
evaluation procedures as a result of attending the institute. Some 
of the negative comments made by two or more participants included: 
Beginners were lost and frustrated by too much theory 
More opportunities needed for participants to get acquainted 
Much information presented was of little practical value 
Better travel arrangements are needed 
Select group leaders who have more experience 
More time was needed for individual and small group work 
Identify participants earlier and supply with advance 
information about program 
Have more social activities for participants 

Some of the positive comments offered by two or more participants 
included : 

Hospitality was excellent 

Meeting facilities were superior 

Staff should be commended for fine effort 

Excellent exchange of ideas and problems 

Outstanding selection of participants 

Very well planned and conducted 

Appreciated strict adherence to schedule 

A fine institute - staff and directors worked hard 

Sumnn * of Plans of Action 

One of the expected outcomes of the institute was that each 
participant or group of participants (from a particular institution or 
state) would develop a tentative ”plan of action” or statement of intent 
which had relevance for his or their work at home bas»3. Even though the 
plans developed were tentative, they were to be realistic and a serious 
attempt made to implement them. Because of the wide diversity of these 
plans, the summary which follows had to be highly subjective and genera] 
in nature. 

A copy of the plan of action assignment given each participant 
may be seen in Appendix E. The assignment suggested developing one of 
three possible projects: 

(a) Outlining procedures for implementing a viable local or 
statewide evaluation system 

(b) Planning an irfService training program for further disseminat- 
ing the knowledges, skills, and materials obtained 

(c) Outlining procedures for conducting a local or statewide 
follow-up system 
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It was recommended that the plan include a statement of the present 
situation with regard to evaluation, a statement of the desired situation, 
and the procedures to be followed to accomplish the desired situation. 
Three hours of regular institute time were allotted for development of 
the plans which were submitted to the institute staff on the last uay 
of the conference. 

Eighty-five of the ninty-two participants submitted an individual 
plan of action or participated with a group in the development of such. 
Seventy-four persons completed their plans while at the institute and 
eleven persons sent their plan to the institute director at a later date. 

A few of the group leader? and recorders, because of thei:: responsibility 
for developing a group report, did not have time to prepare an individual 
plan. Two persons reported turning in plans which were never located. 

Many of the participants chose the option of working together with 
other persons from their state. A total of 49 different plans of action 
were submitted by the 85 participants. Thirty of the plans were 
individually developed and the other 19 plans were cooperatively 
developed by 55 participants working mostly in teams of two or three 
persons. In several instances, everyone from the same state, regardless 
of their level of responsibility, worked together. The plans developed 
by these groups were of a general nature and without the close adaptation 
to individual situations which was desired. These groups were desirable, 
however, from the standpoint of providing local and state leaders an 
opportunity to interact and cooperatively plan an evaluation approach 
more acceptable ^o all. 

As called for in the instructions, nearly all of the plans 
contained a section describing the present status of evaluation in their 
school or state. Participants in nearly half of the plans indicated 
there was no significant program evaluation effort of any type taking 
place. Some typical comments included: 

"Program evaluation is limited to what the state supervisors do 

on their periodic visits" 

"Our evaluation procedures are not well defined" 

"We have no standard system of evaluation at the present time." 

In fifteen of the plans, the authors indicated that some type of process 
and/or product evaluation was being conducted. Emphasis in the past 
appeared to center on the ui»e of evaluative criteria instruments on 
a self-analysis basis, but some had recently initiated or were in the 
process of developing a student follow-up system. Ten of the plans 
developed did not contain any description of the present situation. 

Another way in which the plans were analyzed was in terms of the 
type of program proposed. Most of the plans, 41 of the 49, described 
what might be labelled a fairly comprehensive approach to evaluating 
local or state programs of vocational and technical education. The 
comprehensive programs described in most cases included specific 
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procedures for evaluating both the program process and the program 
product:. General and/or specific objectives of the evaluation effort 
were stated in many of the plans. 



Some of 
1 . 
2 . 



3. 



4. 



5. 



the most common objectives listed were as follows: 

To evaluate existing programs in terms of stated objectives 
To measure the effectiveness and efficiency of vocational 
education programs and services 

To make recommendations for program improvement and program 
expansion 

To determine the extent to which programs are meeting student 
and manpower needs 

To develop state and local leadership competencies 



About one-half of the comprehensive plans, 21 of the 41, included 
plans for conducting some type of evaluation training program. Most 
of these were workshops for local teachers, directors or other vocational 
program administrators. A considerable number of training programs were 
planned for members of state staffs and teacher educators. 



A relatively small number of the plans, 8 of the 49, outlined 
procedures for conducting a specific type of evaluation or for evaluating 
a specific program. Activities proposed included two plans for conducting 
a follow-up study, two for making a cost-benefit analysis, and one 
calling for the development of evaluative criteria instruments for each 
vocational service area. Specific programs to be evaluated included 
an inservice teacher training program and one pre-»service teacher 
education program. One participant, a vocational director for a new 
area school, had no programs in operation but was planning to make 
evaluation a part of the program development process. 

Instructions for the plan of action assignment included the 
suggestion that a tentative time schedule of activities be developed. 
Twenty-five of the 49 plans submitted included some type of activities 
schedule. Most participants planned activities covering a one-year 
period, but a few projected their plans over a two- or three-year period. 

In summary, tentative plans of action were developed by most of 
the participants. They ranged in length from one page to six pages, and 
in quality from high to low. Most of the plans outlined a fairly 
comprehensive evaluation program, while a few were concerned with only 
one type of activity. Considering the amount of time allotted for 
plan development, most of the participants did a very acceptable job. 

For the interested reader, three sample plans modified only to make them 
unidentifiable are included in Appendix E. 



Institute Follow-up 

On April 3, 1970, eight months after the institute was held, a 
brief follow-up form was sent to each of the ninty-two participants 
for the purpose of determining how they had modified and/or expanded 
their evaluation activities since the institute. Participants were 
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also asked to indicate whether they had conducted or assisted with 
training programs on evaluation since the institute, and to indicate 
the extent to which they had been able to initiate or complete 
activities outlined in their plans of action. It was beli >ved the 
best measure of the effectiveness of the institute would be indicated by 
its outcomes-- the future activities of its participants. 

The authors wish to point out their awareness that undoubtedly 
seme of the evaluation activities reported by the participants were a 
direct result of the institute, while at the same time others resulted 
in part from the institute and in part from other factors. Lt is also 
likely that the institute may have had little or no influence on some of 
the activities reported. No attempt was made to determine the degree of 
influence the institute had on the activities reported* In explaining 
the findings, the tacit assumption has been made that the institute 
probably inf^ienced to some extent all the activities reported. 

A cover letter and pre-addressed stamped envelope accompanied the 
follow-up form. To help the participant in responding to the question 
about his plan of action, a copy of the plan he developed or helped 
develop was also mailed. A copy of the follow-up form used may be seen 
in Appendix D. 

Twenty-four of the 32 trainees classified as having local or 
regional responsibilities returned their follow-up questionnaire for a 
73% response. Of the 60 participants categorized as having state or 
national responsibilities, 44 returned their forms for a 73.3% 
return. The overall response rate was 73.9%, with 68 out of the 92 
trainees responding. From a desire to assure trainees of as much 
confiderTialness as possible, the questionnaires were not coded and did 
not require the signature of the respondent. It was fe?t that coding 
and/or requiring a signature would have an inhibiting effect on the 
honesty and frankness of responses. Although most did sign their form, 
several did not, making positive identification of non-respondents 
impossible. Therefore, one mailing only of the questionnaire was used. 
Returns of 70 to 80% are generally considered characteristic of 
reputable educational research, and sufficient to eliminate the effect 
of non-response bias. 

Since it was felt that the type of activities initiated and/ox 
modified might depend to a considerable degree upon the participants’ 
level of responsibility, the results of the follow-up are reported in 
three ways. First, the responses of persons working at the local or 
regional level are presented and discussed. These results are 
followed by a summary of the responses of persons working at the 
state or national level, and finally by a summary of all participant 
responses . 

The first question on the follow-up asked ’’Which of the following 
evaluation activities, if any, have you modified and/or initiated since 
your participation in the institute?” A list of eleven Activities was 
developed after a careful review of the plans of action submitted by the 
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participants. Respondents were asked to check as many of the items in 
each column as was appropriate. Table III presents a numerical summary 
of the responses given by participants having local or regional 
responsibilities . 



TABLE III 

FREQUENCY OF EVALUATION ACTIVITIES MODIFIED AND/OR 
INITIATE AS REPORTED BY TWENTY -FOUR PARTICIPANTS 
HAVING LOCAL OR REGIONAL RESPONSIBILITIES 



Activity 


No. Reporting 
Activity Modified 


No. Reporting 
Activity Initiated 


Program philosophy and objectives 


7 


10 


Citizen evaluation committee (s) 


3 


10 


Student follow-up survey 


7 


9 


Staff evaluation committee (s) 


9 


9 


Self-evaluation procedures 


5 


9 


Program evaluative criteria 


9 


8 


Manpower needs survey 


5 


8 


State evaluation guidelines 


1 


7 


Visiting team evaluations 


4 


5 


Employer satisfaction survey 


4 


4 


Cos t-eff ectiveness study 


2 


3 


Other 


1 


1 



A quick glance at Table III indicates that these trainees were 
involved in modifying and/or initiating a large number of evaluation 
activities. The five major activities initiated by participants since 
the institute include: (a) developing program philosophy and objectives, 
(b) i ing citizen evaluation committees, (c) using staff evaluation 
committees, (d) conducting student follow-up surveys, and (e) using 
self-evaluation procedures. 

A large number of persons were also involved with the initiation 
of manpower needs surveys, the development of program evaluative criteria, 
and the establishment of state evaluation guidelines. A few mentioned 
that they as local-level persons were not responsible for establishing 
state evaluation guidelines ' t that the> had actively participated 
in their development. 

Fewer activities were reported as having been modified by these 
participants. One reason for this may be inferred from reviewing the 
plans of action which contained a description of their present situation. 
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In many instances, local and regional level personnel stated that little 
and in some cases nothing in the way of evaluation was being done. Those 
having evaluation activities underway reported the most modification 
being made in student follow-up surveys, program evaluative criteria, 
program philosophy and objectives, and in the use of staff evaluation 
committees . 



TABLE IV 

FREQUENCY OF EVALUATION ACTIVITIES MODIFIED AND/OR 
INITIATED AS REPORTED BY FOURTY-FOUR PARTICIPANTS 
HAVING STATE OR NATIONAL RESPONSIBILITIES 



Activity No. Reporting No. Reporting 

Activity Modified Activity Initiated 



Program evaluative criteria 
State evaluation guidelines 
Student follow-up survey 
Program philosophy and objectives 
Staff evaluation committee(s) 
Visiting team evaluations 
Self-evaluation procedures 
Manpower needs survey 
Citizen evaluation committee (s) 
Cost-effectiveness study 
Employer satisfaction survey 
Other 



15 


18 


13 


18 


16 


16 


14 


16 


9 


15 


8 


14 


7 


14 


9 


13 


4 


10 


3 


8 


5 


6 


1 


3 



Table IV presents a list of the activities and the frequency with 
which they were modified and/or initiated by participants having state 
or national responsibilities. A comparison with Table III indicates 
that these persons were involved to a greater extent with the initiation 
of activities as compared to modifying activities already underway. All 
but two of the activities listed were initiated by ten or more persons 
working at the state or national level. Two activities, the development 
of program evaluative criteria -*\d the development of state evaluation 
guidelines, were initiated by t 0 hteen persons. Two other evaluation 
activities, conducting a student follow-up survey and developing program 
philosophy and objectives were started by sixteen persons. 

A large number of activities were also modified by this group of 
participants since attending the institute. The activity reported as 
being modified more times than any other was that of conducting a student 
follow-up survey. This same activity was tied tor top ranking among 
those activities reported as modified by local and regional level 
persons. Other activities shown by Table IV as being modified most 



0 




48 



frequently include: program evaluative criteria, program philosophy 
and objecti\'es, and state evaluation guidelines. 

A summary of the responses of all participants to the question of 
which evaluation activities have you modified and/or initiated is presented 
in Table V. Considerably more activities were reported as being 
initiated as contrasted with the number of activities being modified . 

This response may well be a reflection of the fact, also reported in the 
plans of action developed, that not much was being done in the way of 
evaluation at any level in some states. On the other hand, a considerable 
number of participants reported they had made modifications in some of 
their evaluation activities. It may be inferred that some, and probably 
most, of the participants learned of new activities they could initiate 
as well as ways to improve their present procedures, while attending the 
institute. 



TABLE V 

FREQUENCY OF EVALUATION ACTIVITIES MODIFIED 
AND/OR INITIATED AS REPORTED BY SIXTY-EIGHT 
PARTICIPANTS AT ALL LEVELS OF RESPONSIBILITY 



Activity 


No. Reporting 
Activity Modified 


Ho. Reporting 
Activity Initiate 


Program evaluative criteria 


24 


26 


Program philosophy and objectives 


21 


26 


Student follow-up survey 


25 


25 


State evaluation guidelines 




25 


Staff evaluation committee (s) 


16 


24 


Self-evaluation procedures 


12 


23 


Manpower needs survey 


14 


21 


Citizen evaluation committee (s) 


7 


20 


Visiting team evaluations 


12 


19 


Cr -effectiveness study 


5 


11 


Employer satisfaction survey 


9 


10 


Other 


2 


4 



Activities reported as modified by twenty or more of all the 
participants included: student follow-up survey, program evaluative 
criteria, and program philosophy and objectives. The two activities 
reported modified the least number of times were cost-effectiveness 
studies and use of citizc ^valuation committee (s) . 

A look at Table V indicates that most of the activities listed 
were initiated by over twenty participants. Evaluation activities 
receiving the most attention were as follows: program evaluative 
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criteria, program philosophy and objectives, student follow-up surveys, 
and state evaluation guidelines. 

Perhaps just as significant as the activities receiving the most 
attention is the fact that the same two activities, cost-effectiveness 
studies and employer satisfaction surveys, were initiated fewer times 
than the other activities by both groups of participants. The complexity 
and early stage of development may explain the low level of interest 
in cost-effectiveness studies. Over twice as many such studies were 
being initiated by state level personnel as by local level personnel. 

Also significant is the fact every participant indicated he 
had been involved in modifying or initiating at least one evaluation 
activity since the institute. Most reported involvement with several 
activities. 

The second question on the follow-up requested information on the 
type of evaluation training programs, if any, that participants had 
conducted or assisted with since the institute. The intent of this 
question was to determine to what extent participants had shared with 
others the knowledges and skills obtained at the institute. Of the 
twenty-four local and regional level trainees, eight report d 
involvement with training programs on evaluation. Two of the 
respondents had been involved with two programs each. All the training 
programs were of an inservice nature involving teachers, counselors, 
and/or adminis trators . 

The length of training program was reported in ter^s of hours or 
days by all but two of the respondents. One respondent indicated ten 
persons were involved in a ''continuing program and another replied that 
"numerous committee meetings" had been held. The length of other 
training programs varied from three hours to two weeks with the 
average program being about three days in length. 

The number receiving training from participants having local 
or regional responsibilities varied from 10 persons to 150 persons. 

The totil number reported involved in such programs was 448. 

Participants working at the state or national level were 
involved to a greater extent than those working at the local level in 
conducting training programs. While only one- third of the local 
or regional persons were involved, considerably more than half of 
the 44 state and national participants had worked with such programs. 

Five of the 26 had worked with two of more training programs. Most 
of the programs were of an inservice nature involving either state 
staff personnel and/or local vocational directors and administrators. 

Iwo vocational teacher educators reported teaching graduate classes on 
program evaluation, and one indicated four weeks of a graduate course 
were devoted to product evaluation. 

Excluding the graduate classes, the length of these programs 
varied from one hour to six days with most being one or two days in 
length. 
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The total number reported to have received some evaluation training 
was 2221. The number trained per program ranged from 6 persons to 
600. Most programs involved training groups of 30 to 40 persons. 

One-half of the 68 participants responding to the follow-up 
indicated they were involved with one or more evaluation training pro- 
grams. These programs involved a total of 2669 persons in programs of 
varying length. Most programs were of an inservice nature involving 
workshops and conferences for vocational teachers, administrators, and 
state staff personnel. This type of involvement by participants in an 
eight month period since the institute provided much opportunity for 
the^e persons to further disseminate the knowledges and skills they had 
obtained . 

The third question on the institute follow-up was designed to 
provide some measure of the extent to which the participants wh*) 
developed a tentative "plan of action" while at the institute were 
able to return to home base and actually initiate or complete the 
activities they had outlined. The reader is reminded at this point 
that many of the plans of action developed were quite comprehensive, 
including a large number of planned activities, while a few others 
were brief, specifying only one or two activities. Another factor 
influencing responses to this question was the fact that some developed 
a time schedule of activities which extended over a period of one or 
two years. Since the follow-up was conducted eight months after the 
institute, there was insufficient time for many to complete their 
planned schedule of activities. To help the respondent be as objective 
as possible in replying to this question, a copy of each participant’s 
individual plan or the plan which he cooperatively developed with 
other participants was mailed with the follow-up form. 

Table VI shows how the participants responded to question three. 

A majority of both groups of participants reported that they had 
initiated or completed "some" of the activities outlined in their plans 
of action. Only one person in the case of the local and regional level 
participants and two persons in the case of state and national level 
participants, indicated that "none" of the planned activities had been 
carried out. Two of the latter explained their lack of activity. One 
trainee commented "my area of responsibility has been changed" and 
the other indicated the evaluation program in the state had "very low 
priority" and no assigned staff. 
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TABLE VI 



SUMMARY OF THE EXTENT TO WHICH PARTICIPANTS 
HAVE BEEN ABLE TO INITIATE OP COMPLETE THE 
ACTIVITIES OUTLINED IN THEIR PLANS OF ACTION 



Extent 
Activities 
Initiated or 
Completed 


Local and 
Regional 
Parties pants 


State and 

National 

Participants 


All 

Participants 


No. 


Percent 


No. 


Percent 


No. 


Percent 


None 


i 


5 


2 


4.9 


3 


4.9 


Some 


12 


60 


26 


63.4 


38 


62.3 


Most 


7 


35 


10 


24.4 


17 


27.9 


All 


0 


0 


3 


7.3 


3 


4.9 


No Response 


4 


16.3 


3 


6.8 


7 


10.3 



A fair percentage, 27.9, of all the participants reported they had 
been able to initiate or complete ,, most ,, of the activities planned. 

None ot the local level participants and only three of the state 
level participants indicated “all" of their planned activities had 
been carried out. Seven of the participants did not respond to 
this question. Some of these persons had served as group leaders and 
recorders during the institute and as a result of these responsibilities, 
did not have time to prepare and submit a plan of action. Two persons 
indicated a plan had been prepared and submitted but copies of them 
could not be located. Although a higher level of completion is 
desirable, in view of the comprehensiveness of most of the plans and 
the longer than eight months time schedule required to implement some 
of the activities, it was concluded that satisfactory progress was 
being made by most of the participants. 

The last item on the follow-up solicited comments. Some 
participants responded with positive comments about the institute, 
but most were of a neutral vein describing their experiences and 
activities since the institute. Many reported they were involved or 
responsible for developing an effective system of evaluation for their 
school or state. The following responses were selected from the many 
comments made. 

- "Institute provided valuable background information and 
experience - materials received there have formed basis for 
statewide evaluation of vocational education" 

- "One of my graduate students has developed a model for evaluating 
area vocational programs - the model is being tried out and the 
study should be completed this summer." 
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“ "Institute has proved to be very helpful to me and to our 
district in re-evaluating our position and developing a more 
effective system." 

- "I have been able to use many of the techniques learned at the 
conference to improve, implement, and further develop the evaluation 
activities which relate to programs under my supervision. Major 
thrusts have been made in the arer> of product evaluation." 

- "Product evaluation, though simple in concept, is difficult to 
implement. However, I*m convinced more and more that the evaluation 
we conduct should begin with our product." 

- "The Interim Council of NCA has now endorsed "product evaluation" 
as one effective means of ascertaining quality in occupational 
education programs." 
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IV. REPORTS, CONCLUSIONS, AND RECOMMENDATIONS 



Reports 

Besides providing the participants with additional knowledges 
and skills for improving vocational education evaluation, the institute 
was planned to result in three other outcomes. One important outcome, 
the development of plans of action, was described in the previous chapter. 
The other two outcomes, development of task group reports and the devel- 
opment of a general guide for improving vocational education evaluation 
are described here. 

The task force groups were established with two major objectives 
in mind. One objective was to provide a setting where each trainee 
would have an opportunity to interact with other participants in a tho- 
rough review and synthesis of the papers presented in such a way as to 
obtain group consensus, where possible, as to the best strategies and 
techniques of program evaluation available. A copy of the complete task 
group assignment may be seen in Appendix E . 

Four major areas of concern or approaches to evaluation were iden- 
tified by the program planning committee. They were as follows; 

(a) State directed evaluation of statewide programs 

(b) State directed evaluation of local programs 

(c) State assisted evaluation of local programs 

(d) Locally directed evaluation of local programs 

The eight task force groups were organized around these areas of con- 
cern with the assignment of participants being made according to their 
preference. A leader and a recorder selected from the participants was 
appointed to provide leadership to each group. The leaders and record- 
ers were given special instructions and an opportunity to ask questions 
at a meeting held the evening before the opening of the institute. 

Daily staff meetings also provided an opportunity to check on the pro- 
gress each group was making and to provide new inputs when needed. A 
considerable amount of time was devoted to the task group sessions and a 
representative of each group was called upon to make an oral report at 
the general session on Friday morning. The full text of each report 
may be found in Appendix G. 

The final outcome sought was the development of a general guide on 
strategies and procedures that could be used by others responsible for 
evaluating programs. The task force reports as well as the original 
papers presented were the major inputs used by the institute staff in 
developing this guide. By design, and reflecting the state of the art, 
the guide is brief and general in nature. A copy of the guide is pre- 
sented in Appendix H. 
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Conclusions 



The institute was planned so as to focus on four major purposes 
or expected outcomes. They were as follows: 

1. Participants would be provided with additional knowledges and 
skills needed for improving vocational education evaluation. 

2. Participants would initiate the development of an evaluation 
plan or scheme which had relevance for use within their agency 
or organization. 

3. Task force groups would review and synthesize the papers pre- 
sented in order to recommend the best or most premising ap- 
proaches and techniques of evaluation available. 

4. The institute staff using the task group reports as well as 
other available inputs would prepare a general guide on 
strategies and procedures of evaluation. 

After analyzing participant satisfaction with the total program, 
reviewing the plans of action developed and the task group reports 
prepared, and studying the results of the follow-up survey, it is con- 
cluded that the Institute for Improving Vocational Education Evaluation 
was successful in accomplishing its major purposes. The focus of the 
institute evaluation effort was on outcomes: the future activities 

of the participants rather than on the process and instructional objec- 
tives of the institute, although general satisfaction with the program 
organization and instructional content was indicated by scores on the 
Institute Evaluation Scale. Responses on this scale indicated that 
most of the participants either agreed or strongly agreed ^hat an 
institute of this nature should be offered again and that the informa- 
tion presented would be useful to them as vocational educators. ihe 
major criticism indica' ^d was that the information presented was too 
theoretical. 

In terms of immediate outcomes, the participants either individ- 
ually or cooperatively developed an evaluation plan relevant to their 
responsibilities at home base. Responses on the follow-up survey indi- 
cated that in most caes seme of the activities planned have been ini- 
tiated or completed since the institute. In fewer cases, most of the 
activities planned had been carried out. From all indications, most 
participants made a serious attempt to implement the activities planned. 
In view of the comprehensiveness of most plans and the limited time 
available, it was concluded that satisfactory progress was being made 
by most participants. 

Responses on the follow-up survey also indicated that the parti- 
cipants had been extensively involved, with few exceptions, in modi- 
fying and/or initiating a wide variety of specific evaluation activi- 
ties. Several comments made on the follow-up survey indicated that the 
skills and materials obtained while at the institute were of consider- 
able value to them as they conducted these activities. 
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